{"cells":[{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"845368C52E49432B8D8076F774D813B0","mdEditEnable":false,"trusted":true},"source":"# 电商产品评论数据情感分析\n\n\n针对用户在电商平台上留下的评论数据，对其进行分词、词性标注和去除停用词等文本预处理。基于预处理后的数据进行情感分析，并使用LDA主题模型提取评论关键信息，以了解用户的需求、意见、购买原因及产品的优缺点等，最终提出改善产品的建议\n\n----\n\n## 数据预处理\n\n### 评论去重\n\n一些电商平台为了避免一些客户长时间不进行评论，往往会设置一道程序，如果用户超过规定的时间仍然没有做出评论，系统就会自动替客户做出评论，这类数据显然没有任何分析价值。由语言的特点可知，在大多数情况下，不同购买者之间的有价值的评论是不会出现完全重复的，如果不同购物者的评论完全重复，那么这些评论一般都是毫无意义的。为了存留更多的有用语料，本节针对完全重复的语料下手，仅删除完全重复部分，以确保保留有用的文本评论信息。"},{"cell_type":"code","execution_count":1,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"DAF242D3C9D94DFA9F93089E593436D2","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport re\nimport jieba.posseg as psg\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline\n\npath = '/home/kesci/input/emotion_analysi7147'"},{"cell_type":"code","execution_count":2,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"A8EF726DF030469B90396BAE81897FDF","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"reviews = pd.read_csv(path+'/reviews.csv')"},{"cell_type":"code","execution_count":3,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"4DC0EEE5C2BD45E88F2BDFB391F57B04","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"stream","text":"(2000, 5)\n","name":"stdout"},{"output_type":"execute_result","metadata":{},"data":{"text/plain":"                                             content         creationTime  \\\n0  东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...  2017-04-17 13:01:54   \n1                               安装师傅很给力，热水器也好用，感谢美的。  2017-04-17 10:45:33   \n2                                          还没安装，基本满意  2017-04-17 10:58:33   \n3  电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...  2017-10-18 20:22:33   \n4  用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...  2017-04-17 09:19:16   \n\n  nickname                            referenceName content_type  \n0    鑫***辰  美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          pos  \n1    切***药  美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          pos  \n2    j***x  美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          pos  \n3    j***2  美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          pos  \n4    j***6  美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          pos  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>content</th>\n      <th>creationTime</th>\n      <th>nickname</th>\n      <th>referenceName</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...</td>\n      <td>2017-04-17 13:01:54</td>\n      <td>鑫***辰</td>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>安装师傅很给力，热水器也好用，感谢美的。</td>\n      <td>2017-04-17 10:45:33</td>\n      <td>切***药</td>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>还没安装，基本满意</td>\n      <td>2017-04-17 10:58:33</td>\n      <td>j***x</td>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...</td>\n      <td>2017-10-18 20:22:33</td>\n      <td>j***2</td>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...</td>\n      <td>2017-04-17 09:19:16</td>\n      <td>j***6</td>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"print(reviews.shape)\nreviews.head()"},{"cell_type":"code","execution_count":4,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C5BDAABAFA5C43158E206E573D3DD74E","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 删除数据记录中所有列值相同的记录\nreviews = reviews[['content','content_type']].drop_duplicates()\ncontent = reviews['content']"},{"cell_type":"code","execution_count":5,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"1DB9F88E834C4C5A83D9665E16390FE5","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"(1974, 2)"},"transient":{}}],"source":"reviews.shape"},{"metadata":{"id":"67CD3A26EDB543D186F52122DC0BBE56","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"                                                content content_type\n0     东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...          pos\n1                                  安装师傅很给力，热水器也好用，感谢美的。          pos\n2                                             还没安装，基本满意          pos\n3     电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...          pos\n4     用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...          pos\n5                        物美价廉啊，特别划算的，而且加热速度快。家里用着不错特别方便          pos\n6                                        价格合理，配置挺高，物美价值          pos\n7                                老师按装是快的，装修中。没试。希望是正常的！          pos\n8     五分是习惯\\r\\n送的快，装的也快，很好，不过装修进行中，为了吊顶只装了热水器，其它后话吧。...          pos\n9     安装的小哥非常好，工作很尽心，我们家是老房子，安装比一般家要费力。安装的非常不错。因为整栋楼...          pos\n10    前天下单买的今天就用上了，目前没有发现问题，比街上卖的便宜很多，店家都说和网上比他家贵很多，...          pos\n11                          冲着3000元以内1.5变频1级能效，制冷效果还不错。          pos\n12    很好以后还在京东购物，我去别的电器问我这型号，说没有我这个型号，有相识的，1299元\\r\\n...          pos\n13    物流非常快，早上送来的，中午就可以安装了，安装时候也非常认真安装师傅非常认真，负责任，热水器...          pos\n14          各方面都很满意，从价格，产品质量，货运，安装调试，工作人员服务态度方面，还有售后服务。          pos\n15                              挺好的，收费安装速度很快，管道走的也整齐，不错          pos\n16                     用了好几天了，才想起来晒单，差点忘了，这款热水器很好，我很喜欢?          pos\n17                             已经安装但还没用 看上去这样 不多做评价 凑字的          pos\n18                                  非常不错，安装人员也很专业，值得信赖！          pos\n19                                 给家里买的 挺好用的 没在家 安装的很好          pos\n20                           之前装的太阳能。坏掉了，现在换了电热水器还方便一些。          pos\n21                           真心感觉不错，装好当晚三个人冲凉还有热水没用完，好评          pos\n22                                真心不错！很喜欢，大品牌！！！啊赞一个！！          pos\n23                                      已经安装好了，希望没什么问题。          pos\n24                                   就是这么完美。以后不用天天等太阳了。          pos\n25                                      美的售后好，预约安装挺方便的！          pos\n26                                      质量不错，暖水挺快的，支持京东          pos\n27                                              不错，物超所值          pos\n28    凌晨一点下单，14点多送到，17点员工上门安装，速度效率扛扛的！特别是赞扬安装师傅，特别专业...          pos\n29    东西不错，价格很实惠，试了下，加热很快，还省电，老爸说看起来很大气，遥控挺灵敏的，客服很负责...          pos\n...                                                 ...          ...\n1970                              这样安装居然收我130的安装费，差评，差评          neg\n1971                                       喷水速度有时快，有时慢！          neg\n1972       价格便宜，可是上门安装的师傅太坑人了，收了差不多400元的安装费，热水器才买了800元，          neg\n1973            美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)          neg\n1974  美的太垃圾了！买到时主板坏的！出厂都没检测吗！质量太差！换了一台！安装完又漏水！我都晕！这就...          neg\n1975  产品有严重的质量问题，无线遥控根本就不能用，这款产品为什么叫无线遥控？Media的产品做的太...          neg\n1976  这款热水器已经是买的第二个了，新机加热是挺快的，只是有时候温度显示不准，遥控器是垃圾级别，要...          neg\n1977  安装师傅太坑人，我铁管上明明有一个开关，还给我接一个开关，也没问我要不要，直接安上了，安点管...          neg\n1978                                              感觉被骗了          neg\n1979  包装是指帮你挂上去，其他费用自己管哈。材料费一百五，一个洗澡房改个电也一百五。总下来就用了1...          neg\n1980                        目前还没有安装，机身有凹陷，由于不方便退换，所以就这样          neg\n1981                                           还不够一个人洗的          neg\n1982  刚买第二天价格就一天一变，请问你们的价保三十天呢？上门安装还收材料费？这就是京东的服务嘛！最...          neg\n1983  差评    买的是带遥控的    但是没哟遥控   联系客服没有任何回应       很是失...          neg\n1984                              一点都不好用，热水量小，也不加热，很差很差          neg\n1985                                    质量太差了，新买的电脑板出问题          neg\n1986                                             不说了，差评          neg\n1987  这个电热水器才900元，但是不包安装费！\\r\\n随便两个接头和两根管就收了150！\\r\\n贵...          neg\n1988  新手，在京东上看到就买了，结果就吃亏了，这是1500w的，都快被市场淘汰了，京东还放在这推销...          neg\n1989                          不是免费安装，花了我315元的费用，真是不可理解！          neg\n1990                                      商品还可以，就是售后太差劲          neg\n1991  什么鸟玩意，写的京东负责售后还要自己联系厂家预约安装，也不知道下单时候填的预约是干嘛的，说好...          neg\n1992  新买的热水器 降温按钮没用，京东厂家来回打电话，一会换显示板 一会换机器，又要拆又要等，算了...          neg\n1993                          之前客服说里面带安装材料，什么都没有，一边高一边低          neg\n1994                                   为什么不给用不锈钢软管啊，，差评          neg\n1995                                差评，差的一塌糊涂，千万别买，上当了，          neg\n1996  热水器还没有安装，就搞一肚子气，安装人员今天推明天，明天推后天，售后安装服务太差，给差评，目...          neg\n1997                                        好不容易网购一下还漏电          neg\n1998                           东西送的挺快，后期报装2天还没人联系我，售后太差          neg\n1999                               买了两个，送到一个，还有一个至今未送到。          neg\n\n[1974 rows x 2 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>content</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>安装师傅很给力，热水器也好用，感谢美的。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>还没安装，基本满意</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>物美价廉啊，特别划算的，而且加热速度快。家里用着不错特别方便</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>价格合理，配置挺高，物美价值</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>老师按装是快的，装修中。没试。希望是正常的！</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>五分是习惯\\r\\n送的快，装的也快，很好，不过装修进行中，为了吊顶只装了热水器，其它后话吧。...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>安装的小哥非常好，工作很尽心，我们家是老房子，安装比一般家要费力。安装的非常不错。因为整栋楼...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>前天下单买的今天就用上了，目前没有发现问题，比街上卖的便宜很多，店家都说和网上比他家贵很多，...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>冲着3000元以内1.5变频1级能效，制冷效果还不错。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>很好以后还在京东购物，我去别的电器问我这型号，说没有我这个型号，有相识的，1299元\\r\\n...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>物流非常快，早上送来的，中午就可以安装了，安装时候也非常认真安装师傅非常认真，负责任，热水器...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>各方面都很满意，从价格，产品质量，货运，安装调试，工作人员服务态度方面，还有售后服务。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>挺好的，收费安装速度很快，管道走的也整齐，不错</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>用了好几天了，才想起来晒单，差点忘了，这款热水器很好，我很喜欢?</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>已经安装但还没用 看上去这样 不多做评价 凑字的</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>非常不错，安装人员也很专业，值得信赖！</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>给家里买的 挺好用的 没在家 安装的很好</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>之前装的太阳能。坏掉了，现在换了电热水器还方便一些。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>真心感觉不错，装好当晚三个人冲凉还有热水没用完，好评</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>真心不错！很喜欢，大品牌！！！啊赞一个！！</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>23</th>\n      <td>已经安装好了，希望没什么问题。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>就是这么完美。以后不用天天等太阳了。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>25</th>\n      <td>美的售后好，预约安装挺方便的！</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>26</th>\n      <td>质量不错，暖水挺快的，支持京东</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>27</th>\n      <td>不错，物超所值</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>28</th>\n      <td>凌晨一点下单，14点多送到，17点员工上门安装，速度效率扛扛的！特别是赞扬安装师傅，特别专业...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>29</th>\n      <td>东西不错，价格很实惠，试了下，加热很快，还省电，老爸说看起来很大气，遥控挺灵敏的，客服很负责...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>1970</th>\n      <td>这样安装居然收我130的安装费，差评，差评</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1971</th>\n      <td>喷水速度有时快，有时慢！</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1972</th>\n      <td>价格便宜，可是上门安装的师傅太坑人了，收了差不多400元的安装费，热水器才买了800元，</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1973</th>\n      <td>美的（Midea）60升预约洗浴 无线遥控 电热水器 F60-15WB5(Y)</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1974</th>\n      <td>美的太垃圾了！买到时主板坏的！出厂都没检测吗！质量太差！换了一台！安装完又漏水！我都晕！这就...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1975</th>\n      <td>产品有严重的质量问题，无线遥控根本就不能用，这款产品为什么叫无线遥控？Media的产品做的太...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1976</th>\n      <td>这款热水器已经是买的第二个了，新机加热是挺快的，只是有时候温度显示不准，遥控器是垃圾级别，要...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1977</th>\n      <td>安装师傅太坑人，我铁管上明明有一个开关，还给我接一个开关，也没问我要不要，直接安上了，安点管...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1978</th>\n      <td>感觉被骗了</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1979</th>\n      <td>包装是指帮你挂上去，其他费用自己管哈。材料费一百五，一个洗澡房改个电也一百五。总下来就用了1...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1980</th>\n      <td>目前还没有安装，机身有凹陷，由于不方便退换，所以就这样</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1981</th>\n      <td>还不够一个人洗的</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1982</th>\n      <td>刚买第二天价格就一天一变，请问你们的价保三十天呢？上门安装还收材料费？这就是京东的服务嘛！最...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1983</th>\n      <td>差评    买的是带遥控的    但是没哟遥控   联系客服没有任何回应       很是失...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1984</th>\n      <td>一点都不好用，热水量小，也不加热，很差很差</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1985</th>\n      <td>质量太差了，新买的电脑板出问题</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1986</th>\n      <td>不说了，差评</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1987</th>\n      <td>这个电热水器才900元，但是不包安装费！\\r\\n随便两个接头和两根管就收了150！\\r\\n贵...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1988</th>\n      <td>新手，在京东上看到就买了，结果就吃亏了，这是1500w的，都快被市场淘汰了，京东还放在这推销...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1989</th>\n      <td>不是免费安装，花了我315元的费用，真是不可理解！</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1990</th>\n      <td>商品还可以，就是售后太差劲</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1991</th>\n      <td>什么鸟玩意，写的京东负责售后还要自己联系厂家预约安装，也不知道下单时候填的预约是干嘛的，说好...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1992</th>\n      <td>新买的热水器 降温按钮没用，京东厂家来回打电话，一会换显示板 一会换机器，又要拆又要等，算了...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1993</th>\n      <td>之前客服说里面带安装材料，什么都没有，一边高一边低</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1994</th>\n      <td>为什么不给用不锈钢软管啊，，差评</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1995</th>\n      <td>差评，差的一塌糊涂，千万别买，上当了，</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1996</th>\n      <td>热水器还没有安装，就搞一肚子气，安装人员今天推明天，明天推后天，售后安装服务太差，给差评，目...</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1997</th>\n      <td>好不容易网购一下还漏电</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1998</th>\n      <td>东西送的挺快，后期报装2天还没人联系我，售后太差</td>\n      <td>neg</td>\n    </tr>\n    <tr>\n      <th>1999</th>\n      <td>买了两个，送到一个，还有一个至今未送到。</td>\n      <td>neg</td>\n    </tr>\n  </tbody>\n</table>\n<p>1974 rows × 2 columns</p>\n</div>"},"transient":{}}],"source":"reviews","execution_count":6},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"D5E1CCC17A124AEF8648F97549E8FE3F","mdEditEnable":false,"trusted":true},"source":"### 数据清洗\n\n通过人工观察数据发现，评论中夹杂着许多数字与字母，对于本案例的挖掘目标而言，这类数据本身并没有实质性帮助。另外，由于该评论文本数据主要是围绕京东商城中美的电热水器进行评价的，其中“京东”“京东商城”“美的”“热水器”“电热水器”等词出现的频数很大，但是对分析目标并没有什么作用，因此可以在分词之前将这些词去除，对数据进行清洗"},{"cell_type":"code","execution_count":7,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"091545468C01468599895BC0666485E3","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 去除英文、数字、京东、美的、电热水器等词语\nstrinfo = re.compile('[0-9a-zA-Z]|京东|美的|电热水器|热水器|')\ncontent = content.apply(lambda x: strinfo.sub('',x))"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"807802B48AAC4E818C24B89D4B26DC39","mdEditEnable":false,"trusted":true},"source":"### 分词、词性标注、去除停用词\n\n词是文本信息处理的基础环节，是将一个单词序列切分成单个单词的过程。准确地分词可以极大地提高计算机对文本信息的识别和理解能力。相反，不准确的分词将会产生大量的噪声，严重干扰计算机的识别理解能力，并对这些信息的后续处理工作产生较大的影响。中文分词的任务就是把中文的序列切分成有意义的词，即添加合适的词串使得所形成的词串反映句子的本意，中文分词的关键问题为切分歧义的消解和未登录词的识别。\n\n未登录词是指词典中没有登录过的人名、地名、机构名、译名及新词语等。当采用匹配的办法来切分词语时，由于词典中没有登录这些词，会引起自动切分词语的困难。\n\n分词最常用的工作包是jieba分词包，jieba分词是Python写成的一个分词开源库，专门用于中文分词，其有3条基本原理，即实现所采用技术。\n1. 基于Trie树结构实现高效的词图扫描，生成句子中汉字所有可能成词情况所构成的有向无环图（DAG）。\n2. 采用动态规划查找最大概率路径，找出基于词频的最大切分组合。\n3. 对于未登录词，采用HMM模型，使用了Viterbi算法，将中文词汇按照BEMS 4个状态来标记。"},{"cell_type":"code","execution_count":8,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"517383AC34504E6D86A327EC238F1078","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"stream","text":"Building prefix dict from the default dictionary ...\nDumping model to file cache /tmp/jieba.cache\nLoading model cost 0.766 seconds.\nPrefix dict has been built succesfully.\n","name":"stderr"}],"source":"# 分词\nworker = lambda s: [(x.word, x.flag) for x in psg.cut(s)] # 自定义简单分词函数\nseg_word = content.apply(worker)"},{"cell_type":"code","execution_count":9,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E67CC614CA7049AA8B95DC4601A07BFD","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"0    [(东西, ns), (收到, v), (这么久, r), (，, x), (都, d), ...\n1    [(安装, v), (师傅, nr), (很, d), (给, p), (力, n), (，...\n2    [(还, d), (没, v), (安装, v), (，, x), (基本, n), (满意...\n3    [(收到, v), (了, ul), (，, x), (自营, vn), (商品, n), ...\n4    [(用, p), (了, ul), (几次, m), (才, d), (来, v), (评价...\nName: content, dtype: object"},"transient":{}}],"source":"seg_word.head()"},{"cell_type":"code","execution_count":10,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"AFE9036ED9124F7E8DDD4B8843F51AE8","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 将词语转为数据框形式，一列是词，一列是词语所在的句子ID，最后一列是词语在该句子的位置\nn_word = seg_word.apply(lambda x: len(x))  # 每一评论中词的个数\n\nn_content = [[x+1]*y for x,y in zip(list(seg_word.index), list(n_word))]\n\n# 将嵌套的列表展开，作为词所在评论的id\nindex_content = sum(n_content, [])\n\nseg_word = sum(seg_word, [])\n# 词\nword = [x[0] for x in seg_word]\n# 词性\nnature = [x[1] for x in seg_word]\n\ncontent_type = [[x]*y for x,y in zip(list(reviews['content_type']), list(n_word))]\n# 评论类型\ncontent_type = sum(content_type, [])\n\nresult = pd.DataFrame({\"index_content\":index_content, \n                       \"word\":word,\n                       \"nature\":nature,\n                       \"content_type\":content_type})"},{"cell_type":"code","execution_count":11,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"45EE0D70974E4BAE91C874201457C664","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content word nature content_type\n0              1   东西     ns          pos\n1              1   收到      v          pos\n2              1  这么久      r          pos\n3              1    ，      x          pos\n4              1    都      d          pos","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>word</th>\n      <th>nature</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>东西</td>\n      <td>ns</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>收到</td>\n      <td>v</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>这么久</td>\n      <td>r</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>，</td>\n      <td>x</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>都</td>\n      <td>d</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"result.head()"},{"cell_type":"code","execution_count":12,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"8067AA51DDB7422785B80F11E1DD7506","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 删除标点符号\nresult = result[result['nature'] != 'x']  # x表示标点符号\n\n# 删除停用词\nstop_path = open(path+\"/stoplist.txt\", 'r',encoding='UTF-8')\nstop = stop_path.readlines()\nstop = [x.replace('\\n', '') for x in stop]\nword = list(set(word) - set(stop))\nresult = result[result['word'].isin(word)]"},{"cell_type":"code","execution_count":13,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"B6E4BCF72A9044F98ECF8288204BEFA9","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content word nature content_type\n0              1   东西     ns          pos\n1              1   收到      v          pos\n2              1  这么久      r          pos\n5              1    忘      v          pos\n8              1   好评      v          pos","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>word</th>\n      <th>nature</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>东西</td>\n      <td>ns</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>收到</td>\n      <td>v</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>这么久</td>\n      <td>r</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>1</td>\n      <td>忘</td>\n      <td>v</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>1</td>\n      <td>好评</td>\n      <td>v</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"result.head()"},{"cell_type":"code","execution_count":14,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E2D03D88E705446C8362A04CCB3BFFAD","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content word nature content_type  index_word\n0              1   东西     ns          pos           0\n1              1   收到      v          pos           1\n2              1  这么久      r          pos           2\n5              1    忘      v          pos           3\n8              1   好评      v          pos           4","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>word</th>\n      <th>nature</th>\n      <th>content_type</th>\n      <th>index_word</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>东西</td>\n      <td>ns</td>\n      <td>pos</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>收到</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>这么久</td>\n      <td>r</td>\n      <td>pos</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>1</td>\n      <td>忘</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>1</td>\n      <td>好评</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"# 构造各词在对应评论的位置列\nn_word = list(result.groupby(by = ['index_content'])['index_content'].count())\nindex_word = [list(np.arange(0, y)) for y in n_word]\n# 词语在该评论的位置\nindex_word = sum(index_word, [])\n# 合并评论id\nresult['index_word'] = index_word\n\nresult.head()"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"20D4771809DF40BF97EB2A9E6D76F5F7","mdEditEnable":false,"trusted":true},"source":"### 提取含名词的评论\n\n由于本案例的目标是对产品特征的优缺点进行分析，类似“不错，很好的产品”“很不错，继续支持”等评论虽然表达了对产品的情感倾向，但是实际上无法根据这些评论提取出哪些产品特征是用户满意的。评论中只有出现明确的名词，如机构团体及其他专有名词时，才有意义，因此需要对分词后的词语进行词性标注。之后再根据词性将含有名词类的评论提取出来。"},{"cell_type":"code","execution_count":15,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"3F344D8696D641EFAF288D97F019A2E9","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content word nature content_type  index_word\n0              1   东西     ns          pos           0\n1              1   收到      v          pos           1\n2              1  这么久      r          pos           2\n5              1    忘      v          pos           3\n8              1   好评      v          pos           4","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>word</th>\n      <th>nature</th>\n      <th>content_type</th>\n      <th>index_word</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>东西</td>\n      <td>ns</td>\n      <td>pos</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>收到</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>这么久</td>\n      <td>r</td>\n      <td>pos</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>1</td>\n      <td>忘</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>1</td>\n      <td>好评</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"# 提取含有名词类的评论,即词性含有“n”的评论\nind = result[['n' in x for x in result['nature']]]['index_content'].unique()\nresult = result[[x in ind for x in result['index_content']]]\nresult.head()"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"9160F32BE7FC43518DC2AC3BAD654EA8","mdEditEnable":false,"trusted":true},"source":"### 绘制词云\n\n绘制词云查看分词效果，词云会将文本中出现频率较高的“关键词”予以视觉上的突出。首先需要对词语进行词频统计，将词频按照降序排序，选择前100个词，使用wordcloud模块中的WordCloud绘制词云，查看分词效果"},{"cell_type":"code","execution_count":16,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"CF367D33FEF34A23848ED591D0C2D97F","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/CF367D33FEF34A23848ED591D0C2D97F/qau4ilu1yb.png\">"},"transient":{}}],"source":"import matplotlib.pyplot as plt\nfrom wordcloud import WordCloud\n\nfrequencies = result.groupby('word')['word'].count()\nfrequencies = frequencies.sort_values(ascending = False)\nbackgroud_Image=plt.imread(path+'/pl.jpg')\n\n# 自己上传中文字体到kesci\nfont_path = '/home/kesci/work/data/fonts/MSYHL.TTC'\nwordcloud = WordCloud(font_path=font_path, # 设置字体，不设置就会出现乱码\n                      max_words=100,\n                      background_color='white',\n                      mask=backgroud_Image)# 词云形状\n\nmy_wordcloud = wordcloud.fit_words(frequencies)\nplt.imshow(my_wordcloud)\nplt.axis('off') \nplt.show()"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"9A5FEE2739FA42008F9D43459D8F2E23","mdEditEnable":false,"trusted":true},"source":"由图可以看出，对评论数据进行预处理后，分词效果较为符合预期。其中“安装”“师傅”“售后”“物流”“服务”等词出现频率较高，因此可以初步判断用户对产品的这几个方面比较重视"},{"cell_type":"code","execution_count":17,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"A6E8F57F1EF5410FAD171E469E74F761","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 将结果保存\nresult.to_csv(\"./word.csv\", index = False, encoding = 'utf-8')"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"628E687861294D7489906047391A55E2","mdEditEnable":false,"trusted":true},"source":"## 词典匹配\n\n### 评论数据情感倾向分析\n\n匹配情感词情感倾向也称为情感极性。在某商品评论中，可以理解为用户对该商品表达自身观点所持的态度是支持、反对还是中立，即通常所指的正面情感、负面情感、中性情感。由于本案例主要是对产品的优缺点进行分析，因此只要确定用户评论信息中的情感倾向方向分析即可，不需要分析每一评论的情感程度。\n\n对评论情感倾向进行分析首先要对情感词进行匹配，主要采用词典匹配的方法，本案例使用的情感词表是2007年10月22日知网发布的“情感分析用词语集（beta版）”，主要使用“中文正面评价”词表、“中文负面评价”“中文正面情感”“中文负面情感”词表等。将“中文正面评价”“中文正面情感”两个词表合并，并给每个词语赋予初始权重1，作为本案例的正面评论情感词表。将“中文负面评价”“中文负面情感”两个词表合并，并给每个词语赋予初始权重-1，作为本案例的负面评论情感词表。\n\n一般基于词表的情感分析方法，分析的效果往往与情感词表内的词语有较强的相关性，如果情感词表内的词语足够全面，并且词语符合该案例场景下所表达的情感，那么情感分析的效果会更好。针对本案例场景，需要在知网提供的词表基础上进行优化，例如“好评”“超值”“差评”“五分”等词只有在网络购物评论上出现，就可以根据词语的情感倾向添加至对应的情感词表内。将“满意”“好评”“很快”“还好”“还行”“超值”“给力”“支持”“超好”“感谢”“太棒了”“厉害”“挺舒服”“辛苦”“完美”“喜欢”“值得”“省心”等词添加进正面情感词表。将“差评”“贵”“高”“漏水”等词加入负面情感词表。读入正负面评论情感词表，正面词语赋予初始权重1，负面词语赋予初始权重-1。"},{"cell_type":"code","execution_count":18,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"DCD89D9B46074D2B9BEDA3744797C40C","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"     word  weight  index_content nature content_type  index_word\n1459   东西     NaN              1     ns          pos           0\n1683   收到     NaN              1      v          pos           1\n1762  这么久     NaN              1      r          pos           2\n1766    忘     NaN              1      v          pos           3\n390    好评     1.0              1      v          pos           4","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>word</th>\n      <th>weight</th>\n      <th>index_content</th>\n      <th>nature</th>\n      <th>content_type</th>\n      <th>index_word</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1459</th>\n      <td>东西</td>\n      <td>NaN</td>\n      <td>1</td>\n      <td>ns</td>\n      <td>pos</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1683</th>\n      <td>收到</td>\n      <td>NaN</td>\n      <td>1</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1762</th>\n      <td>这么久</td>\n      <td>NaN</td>\n      <td>1</td>\n      <td>r</td>\n      <td>pos</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>1766</th>\n      <td>忘</td>\n      <td>NaN</td>\n      <td>1</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>390</th>\n      <td>好评</td>\n      <td>1.0</td>\n      <td>1</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"word = pd.read_csv(\"./word.csv\")\n\n# 读入正面、负面情感评价词\npos_comment = pd.read_csv(path+\"/正面评价词语（中文）.txt\", header=None,sep=\"\\n\", \n                          encoding = 'utf-8', engine='python')\nneg_comment = pd.read_csv(path+\"/负面评价词语（中文）.txt\", header=None,sep=\"\\n\", \n                          encoding = 'utf-8', engine='python')\npos_emotion = pd.read_csv(path+\"/正面情感词语（中文）.txt\", header=None,sep=\"\\n\", \n                          encoding = 'utf-8', engine='python')\nneg_emotion = pd.read_csv(path+\"/负面情感词语（中文）.txt\", header=None,sep=\"\\n\", \n                          encoding = 'utf-8', engine='python') \n\n# 合并情感词与评价词\npositive = set(pos_comment.iloc[:,0])|set(pos_emotion.iloc[:,0])\nnegative = set(neg_comment.iloc[:,0])|set(neg_emotion.iloc[:,0])\n\n# 正负面情感词表中相同的词语\nintersection = positive&negative\n\npositive = list(positive - intersection)\nnegative = list(negative - intersection)\n\npositive = pd.DataFrame({\"word\":positive,\n                         \"weight\":[1]*len(positive)})\nnegative = pd.DataFrame({\"word\":negative,\n                         \"weight\":[-1]*len(negative)}) \n\nposneg = positive.append(negative)\n\n\n# 将分词结果与正负面情感词表合并，定位情感词\ndata_posneg = posneg.merge(word, left_on = 'word', right_on = 'word', \n                           how = 'right')\ndata_posneg = data_posneg.sort_values(by = ['index_content','index_word'])\n\ndata_posneg.head()"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"5EF6EB9920F0407A9752B36DE21F7629","mdEditEnable":false,"trusted":true},"source":"### 修正情感倾向\n\n情感倾向修正主要根据情感词前面两个位置的词语是否存在否定词而去判断情感值的正确与否，由于汉语中存在多重否定现象，即当否定词出现奇数次时，表示否定意思；当否定词出现偶数次时，表示肯定意思。按照汉语习惯，搜索每个情感词前两个词语，若出现奇数否定词，则调整为相反的情感极性。\n\n本案例使用的否定词表共有19个否定词，分别为：不、没、无、非、莫、弗、毋、未、否、别、無、休、不是、不能、不可、没有、不用、不要、从没、不太。\n\n读入否定词表，对情感值的方向进行修正。计算每条评论的情感得分，将评论分为正面评论和负面评论，并计算情感分析的准确率。"},{"cell_type":"code","execution_count":19,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E71016149DA741CB8D1925CF14620DBE","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 载入否定词表\nnotdict = pd.read_csv(path+\"/not.csv\")\n\n# 构造新列，作为经过否定词修正后的情感值\ndata_posneg['amend_weight'] = data_posneg['weight']\ndata_posneg['id'] = np.arange(0, len(data_posneg))\n\n# 只保留有情感值的词语\nonly_inclination = data_posneg.dropna().reset_index(drop=True)\n\nindex = only_inclination['id']\n\n\nfor i in np.arange(0, len(only_inclination)):\n    # 提取第i个情感词所在的评论\n    review = data_posneg[data_posneg['index_content'] == only_inclination['index_content'][i]]\n    review.index = np.arange(0, len(review))\n    # 第i个情感值在该文档的位置\n    affective = only_inclination['index_word'][i]\n    if affective == 1:\n        ne = sum([i in notdict['term'] for i in review['word'][affective - 1]])%2\n        if ne == 1:\n            data_posneg['amend_weight'][index[i]] = -data_posneg['weight'][index[i]]          \n    elif affective > 1:\n        ne = sum([i in notdict['term'] for i in review['word'][[affective - 1, \n                  affective - 2]]])%2\n        if ne == 1:\n            data_posneg['amend_weight'][index[i]] = -data_posneg['weight'][index[i]]\n            \n\n            \n# 更新只保留情感值的数据\nonly_inclination = only_inclination.dropna()\n\n# 计算每条评论的情感值\nemotional_value = only_inclination.groupby(['index_content'],\n                                           as_index=False)['amend_weight'].sum()\n\n# 去除情感值为0的评论\nemotional_value = emotional_value[emotional_value['amend_weight'] != 0]"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C7A82636137D46E088C2329CF92867F0","mdEditEnable":false,"trusted":true},"source":"### 查看情感分析效果"},{"cell_type":"code","execution_count":20,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"CE3A6B741DBD45108B5DDF8A6138DF22","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content  amend_weight a_type\n0              1           4.0    pos\n1              2           1.0    pos\n2              4           4.0    pos\n3              5           1.0    pos\n4              6           1.0    pos","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>amend_weight</th>\n      <th>a_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>1.0</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>4</td>\n      <td>4.0</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>5</td>\n      <td>1.0</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>6</td>\n      <td>1.0</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"# 给情感值大于0的赋予评论类型（content_type）为pos,小于0的为neg\nemotional_value['a_type'] = ''\nemotional_value['a_type'][emotional_value['amend_weight'] > 0] = 'pos'\nemotional_value['a_type'][emotional_value['amend_weight'] < 0] = 'neg'\n\nemotional_value.head()"},{"cell_type":"code","execution_count":21,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"20DF335F909B45A39704FF8B2EAF564C","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"   index_content  amend_weight a_type word nature content_type  index_word\n0              1           4.0    pos   东西     ns          pos           0\n1              1           4.0    pos   收到      v          pos           1\n2              1           4.0    pos  这么久      r          pos           2\n3              1           4.0    pos    忘      v          pos           3\n4              1           4.0    pos   好评      v          pos           4","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>amend_weight</th>\n      <th>a_type</th>\n      <th>word</th>\n      <th>nature</th>\n      <th>content_type</th>\n      <th>index_word</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n      <td>东西</td>\n      <td>ns</td>\n      <td>pos</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n      <td>收到</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n      <td>这么久</td>\n      <td>r</td>\n      <td>pos</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n      <td>忘</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1</td>\n      <td>4.0</td>\n      <td>pos</td>\n      <td>好评</td>\n      <td>v</td>\n      <td>pos</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"# 查看情感分析结果\nresult = emotional_value.merge(word, \n                               left_on = 'index_content', \n                               right_on = 'index_content',\n                               how = 'left')\nresult.head()"},{"cell_type":"code","execution_count":22,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E15356CF66EF47E08549E223BAF1CC38","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"    index_content content_type a_type\n0               1          pos    pos\n14              2          pos    pos\n18              4          pos    pos\n38              5          pos    pos\n61              6          pos    pos","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index_content</th>\n      <th>content_type</th>\n      <th>a_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>pos</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>2</td>\n      <td>pos</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>4</td>\n      <td>pos</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>38</th>\n      <td>5</td>\n      <td>pos</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>61</th>\n      <td>6</td>\n      <td>pos</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"result = result[['index_content','content_type', 'a_type']].drop_duplicates()\nresult.head()"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"976679C15AAF4943AEF3581CB2E7BF6C","mdEditEnable":false,"trusted":true},"source":"假定用户在评论时不存在“选了好评的标签，而写了差评内容”的情况，比较原评论的评论类型与情感分析得出的评论类型，绘制情感倾向分析混淆矩阵，查看词表的情感分析的准确率。"},{"cell_type":"code","execution_count":23,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"37532DBBB83E44EF83C946A24C3D182B","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"a_type        neg  pos  All\ncontent_type               \nneg           363   55  418\npos            40  443  483\nAll           403  498  901","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th>a_type</th>\n      <th>neg</th>\n      <th>pos</th>\n      <th>All</th>\n    </tr>\n    <tr>\n      <th>content_type</th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>neg</th>\n      <td>363</td>\n      <td>55</td>\n      <td>418</td>\n    </tr>\n    <tr>\n      <th>pos</th>\n      <td>40</td>\n      <td>443</td>\n      <td>483</td>\n    </tr>\n    <tr>\n      <th>All</th>\n      <td>403</td>\n      <td>498</td>\n      <td>901</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"# 交叉表:统计分组频率的特殊透视表\nconfusion_matrix = pd.crosstab(result['content_type'], result['a_type'], \n                               margins=True)\nconfusion_matrix.head()"},{"cell_type":"code","execution_count":24,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F77BFC4DB2994DA49579970D9963ACA6","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"0.8945615982241953"},"transient":{}}],"source":"(confusion_matrix.iat[0,0] + confusion_matrix.iat[1,1])/confusion_matrix.iat[2,2]"},{"cell_type":"code","execution_count":25,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"0BC21841DD474CE5B20FF1A338D3BB96","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 提取正负面评论信息\nind_pos = list(emotional_value[emotional_value['a_type'] == 'pos']['index_content'])\nind_neg = list(emotional_value[emotional_value['a_type'] == 'neg']['index_content'])\nposdata = word[[i in ind_pos for i in word['index_content']]]\nnegdata = word[[i in ind_neg for i in word['index_content']]]"},{"cell_type":"code","execution_count":26,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"D4EC34AD42364E589966F9E7E53EF2D7","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/D4EC34AD42364E589966F9E7E53EF2D7/qau4iodesy.png\">"},"transient":{}},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/D4EC34AD42364E589966F9E7E53EF2D7/qau4ipx2vb.png\">"},"transient":{}}],"source":"# 绘制词云\nimport matplotlib.pyplot as plt\nfrom wordcloud import WordCloud\n\n\n# 正面情感词词云\nfreq_pos = posdata.groupby('word')['word'].count()\nfreq_pos = freq_pos.sort_values(ascending = False)\nbackgroud_Image=plt.imread(path+'/pl.jpg')\nwordcloud = WordCloud(font_path=font_path,\n                      max_words=100,\n                      background_color='white',\n                      mask=backgroud_Image)\npos_wordcloud = wordcloud.fit_words(freq_pos)\nplt.imshow(pos_wordcloud)\nplt.axis('off') \nplt.show()\n\n\n# 负面情感词词云\nfreq_neg = negdata.groupby(by = ['word'])['word'].count()\nfreq_neg = freq_neg.sort_values(ascending = False)\nneg_wordcloud = wordcloud.fit_words(freq_neg)\nplt.imshow(neg_wordcloud)\nplt.axis('off') \nplt.show()"},{"cell_type":"code","execution_count":27,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F8D7051CBA7C47D8AF6C7B6F3D39AE09","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 将结果写出,每条评论作为一行\nposdata.to_csv(\"./posdata.csv\", index = False, encoding = 'utf-8')\nnegdata.to_csv(\"./negdata.csv\", index = False, encoding = 'utf-8')"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"788E70CC6D8A48F087A64BFE518A4D7B","mdEditEnable":false,"trusted":true},"source":"由图正面情感评论词云可知，“不错”“满意”“好评”等正面情感词出现的频数较高，并且没有掺杂负面情感词语，可以看出情感分析能较好地将正面情感评论抽取出来。\n\n由图负面情感评论词云可知，“差评”“垃圾”“不好”“太差”等负面情感词出现的频数较高，并且没有掺杂正面情感词语，可以看出情感分析能较好地将负面情感评论抽取出来。\n\n____\n\n## LinearSVC模型预测情感\n将数据集划分为训练集和测试集(8:2)，通过TfidfVectorizer将评论文本向量化，在来训练LinearSVC模型，查看模型在训练集上的得分，预测测试集"},{"metadata":{"id":"69B48809A5C746B38AB8C069BF8CA0E8","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"                                             content content_type\n0  东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...          pos\n1                               安装师傅很给力，热水器也好用，感谢美的。          pos\n2                                          还没安装，基本满意          pos\n3  电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...          pos\n4  用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...          pos","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>content</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>安装师傅很给力，热水器也好用，感谢美的。</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>还没安装，基本满意</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...</td>\n      <td>pos</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...</td>\n      <td>pos</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"reviews.head()","execution_count":28},{"metadata":{"id":"93B2E83B12D0488D857AEC9C68920374","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"                                             content  content_type\n0  东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...           1.0\n1                               安装师傅很给力，热水器也好用，感谢美的。           1.0\n2                                          还没安装，基本满意           1.0\n3  电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...           1.0\n4  用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...           1.0","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>content</th>\n      <th>content_type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>东西收到这么久，都忘了去好评，美的大品牌，值得信赖，东西整体来看，个人感觉还不错，没有出现什...</td>\n      <td>1.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>安装师傅很给力，热水器也好用，感谢美的。</td>\n      <td>1.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>还没安装，基本满意</td>\n      <td>1.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>电热水器收到了，京东自营商品就是好，发货速度快，品质有保障，安装效果好，宝贝非常喜欢，冬天可...</td>\n      <td>1.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>用了几次才来评价，对产品非常满意，加热快保温时间长，售后服务特别好，主动打电话询问送货情况帮...</td>\n      <td>1.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"transient":{}}],"source":"reviews['content_type'] = reviews['content_type'].map(lambda x:1.0 if x == 'pos' else 0.0)\nreviews.head()","execution_count":30},{"metadata":{"id":"A2C4A8EF064243B2ACD8B29BD54CD263","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"((1579,), (1579,), (395,), (395,))"},"transient":{}}],"source":"from sklearn.feature_extraction.text import TfidfVectorizer as TFIDF  # 原始文本转化为tf-idf的特征矩阵\nfrom sklearn.svm import LinearSVC\nfrom sklearn.calibration import CalibratedClassifierCV\nfrom sklearn.model_selection import train_test_split\n\n# 将有标签的数据集划分成训练集和测试集\ntrain_X,valid_X,train_y,valid_y = train_test_split(reviews['content'],reviews['content_type'],test_size=0.2,random_state=42)\n\ntrain_X.shape,train_y.shape,valid_X.shape,valid_y.shape","execution_count":31},{"metadata":{"id":"6A49D2A4973942C3A216C93FDDEE143D","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[],"source":"# 模型构建\nmodel_tfidf = TFIDF(min_df=5, max_features=5000, ngram_range=(1,3), use_idf=1, smooth_idf=1)\n# 学习idf vector\nmodel_tfidf.fit(train_X)\n# 把文档转换成 X矩阵（该文档中该特征词出现的频次），行是文档个数，列是特征词的个数\ntrain_vec = model_tfidf.transform(train_X)","execution_count":32},{"metadata":{"id":"B664998BDD5B4FDF845FAA21EAF65BC6","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"array([[0.81757399, 0.52795963, 0.22985082, ..., 0.        , 0.        ,\n        0.        ],\n       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n        0.        ],\n       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n        0.        ],\n       ...,\n       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n        0.        ],\n       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n        0.        ],\n       [0.        , 0.        , 0.        , ..., 0.        , 0.        ,\n        0.        ]])"},"transient":{}}],"source":"train_vec.toarray()","execution_count":36},{"metadata":{"id":"21E8D8F5CBD942EF97C246621400E949","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"CalibratedClassifierCV(base_estimator=LinearSVC(C=1.0, class_weight=None,\n                                                dual=True, fit_intercept=True,\n                                                intercept_scaling=1,\n                                                loss='squared_hinge',\n                                                max_iter=1000,\n                                                multi_class='ovr', penalty='l2',\n                                                random_state=None, tol=0.0001,\n                                                verbose=0),\n                       cv='warn', method='sigmoid')"},"transient":{}}],"source":"# 模型训练\nmodel_SVC = LinearSVC()\nclf = CalibratedClassifierCV(model_SVC)\nclf.fit(train_vec,train_y)","execution_count":37},{"metadata":{"id":"169709D893754A9884653124210ECDFB","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"array([[0.58486491, 0.41513509],\n       [0.58486491, 0.41513509],\n       [0.03801183, 0.96198817],\n       [0.10000182, 0.89999818],\n       [0.94225016, 0.05774984]])"},"transient":{}}],"source":"# 把文档转换成矩阵\nvalid_vec = model_tfidf.transform(valid_X)\n# 验证\npre_valid = clf.predict_proba(valid_vec)\npre_valid[:5]","execution_count":39},{"metadata":{"id":"1385EF24AA8A4BCE80BEE684306472F1","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"正例: 76\n负例: 319\n","name":"stdout"}],"source":"pre_valid = clf.predict(valid_vec)\nprint('正例:',sum(pre_valid == 1))\nprint('负例:',sum(pre_valid == 0))","execution_count":40},{"metadata":{"id":"D1C802695DB64320AD7BEA5412EF3FF8","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"trusted":true,"collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"准确率: 0.6683544303797468\n","name":"stdout"}],"source":"from sklearn.metrics import accuracy_score\n\nscore = accuracy_score(pre_valid,valid_y)\nprint(\"准确率:\",score)","execution_count":41},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"D3C95D238755409F8ED65FB96EB72BE9","mdEditEnable":false,"trusted":true},"source":"## LDA模型\n\nLDA是一种文档主题生成模型，包含词、主题和文档三层结构。\n\n1. 主题模型在自然语言处理等领域是用来在一系列文档中发现抽象主题的一种统计模型。判断两个文档相似性的传统方法是通过查看两个文档共同出现的单词的多少，如TF（词频）、TF-IDF（词频—逆向文档频率）等，这种方法没有考虑文字背后的语义关联，例如，两个文档共同出现的单词很少甚至没有，但两个文档是相似的，因此在判断文档相似性时，需要使用主题模型进行语义分析并判断文档相似性。如果一篇文档有多个主题，则一些特定的可代表不同主题的词语就会反复出现，此时，运用主题模型，能够发现文本中使用词语的规律，并且把规律相似的文本联系到一起，以寻求非结构化的文本集中的有用信息。例如，在美的电热水器的商品评论文本数据中，代表电热水器特征的词语如“安装”“出水量”“服务”等会频繁地出现在评论中，运用主题模型，把热水器代表性特征相关的情感描述性词语与对应特征的词语联系起来，从而深入了解用户对电热水器的关注点及用户对于某一特征的情感倾向\n\n\n\n2. LDA主题模型潜在狄利克雷分配，即LDA模型（Latent Dirichlet Allocation，LDA）是由Blei等人在2003年提出的生成式主题模型。所谓生成模型，就是说，我们认为一篇文章的每个词都是通过“以一定概率选择了某个主题，并从这个主题中以一定概率选择某个词语”这样一个过程得到。文档到主题服从多项式分布，主题到词服从多项式分布。LDA模型也被称为3层贝叶斯概率模型，包含文档（d）、主题（z）、词（w）3层结构，能够有效对文本进行建模，和传统的空间向量模型（VSM）相比，增加了概率的信息。通过LDA主题模型，能够挖掘数据集中的潜在主题，进而分析数据集的集中关注点及其相关特征词。LDA模型采用词袋模型（Bag of Words，BOW）将每一篇文档视为一个词频向量，从而将文本信息转化为易于建模的数字信息。定义词表大小为L，一个L维向量（1，0，0，…，0，0）表示一个词。由N个词构成的评论记为d=（w1，w2，…，wN）。假设某一商品的评论集D由M篇评论构成，记为D=（d1，d2，…，dM）。M篇评论分布着K个主题，记为Zi=（i=1，2，…，K）。记a和b为狄利克雷函数的先验参数，q为主题在文档中的多项分布的参数，其服从超参数为a的Dirichlet先验分布，f为词在主题中的多项分布的参数，其服从超参数b的Dirichlet先验分布。"},{"cell_type":"code","execution_count":50,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"4FAECB06C8CB4FFE8012CA732860835C","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"import re\nimport itertools\n\nfrom gensim import corpora, models\n\n\n# 载入情感分析后的数据\nposdata = pd.read_csv(\"./posdata.csv\", encoding = 'utf-8')\nnegdata = pd.read_csv(\"./negdata.csv\", encoding = 'utf-8')\n\n\n# 建立词典\npos_dict = corpora.Dictionary([[i] for i in posdata['word']])  # 正面\nneg_dict = corpora.Dictionary([[i] for i in negdata['word']])  # 负面\n\n# 建立语料库\npos_corpus = [pos_dict.doc2bow(j) for j in [[i] for i in posdata['word']]]  # 正面\nneg_corpus = [neg_dict.doc2bow(j) for j in [[i] for i in negdata['word']]]   # 负面"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"11463888AF7F492F8B5FB992DB614019","mdEditEnable":false,"trusted":true},"source":"### 主题数寻优\n\n基于相似度的自适应最优LDA模型选择方法，确定主题数并进行主题分析。实验证明该方法可以在不需要人工调试主题数目的情况下，用相对少的迭代找到最优的主题结构。\n\n具体步骤如下：\n1. 取初始主题数k值，得到初始模型，计算各主题之间的相似度（平均余弦距离）。\n2. 增加或减少k值，重新训练模型，再次计算各主题之间的相似度。\n3. 重复步骤2直到得到最优k值。"},{"cell_type":"code","execution_count":51,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C34D7F0D2F48443688FE0B27BEDD747C","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 余弦相似度函数\ndef cos(vector1, vector2):\n    dot_product = 0.0;  \n    normA = 0.0;  \n    normB = 0.0;  \n    for a,b in zip(vector1, vector2): \n        dot_product += a*b  \n        normA += a**2  \n        normB += b**2  \n    if normA == 0.0 or normB==0.0:  \n        return(None)  \n    else:  \n        return(dot_product / ((normA*normB)**0.5))   \n\n# 主题数寻优\ndef lda_k(x_corpus, x_dict):  \n    \n    # 初始化平均余弦相似度\n    mean_similarity = []\n    mean_similarity.append(1)\n    \n    # 循环生成主题并计算主题间相似度\n    for i in np.arange(2,11):\n        # LDA模型训练\n        lda = models.LdaModel(x_corpus, num_topics = i, id2word = x_dict)\n        for j in np.arange(i):\n            term = lda.show_topics(num_words = 50)\n            \n        # 提取各主题词\n        top_word = []\n        for k in np.arange(i):\n            top_word.append([''.join(re.findall('\"(.*)\"',i)) \\\n                             for i in term[k][1].split('+')])  # 列出所有词\n           \n        # 构造词频向量\n        word = sum(top_word,[])  # 列出所有的词   \n        unique_word = set(word)  # 去除重复的词\n        \n        # 构造主题词列表，行表示主题号，列表示各主题词\n        mat = []\n        for j in np.arange(i):\n            top_w = top_word[j]\n            mat.append(tuple([top_w.count(k) for k in unique_word]))  \n            \n        p = list(itertools.permutations(list(np.arange(i)),2))\n        l = len(p)\n        top_similarity = [0]\n        for w in np.arange(l):\n            vector1 = mat[p[w][0]]\n            vector2 = mat[p[w][1]]\n            top_similarity.append(cos(vector1, vector2))\n            \n        # 计算平均余弦相似度\n        mean_similarity.append(sum(top_similarity)/l)\n    return(mean_similarity)"},{"cell_type":"code","execution_count":52,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"DA33D34B1A8140088B76FBB67F9E30AC","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# 计算主题平均余弦相似度\npos_k = lda_k(pos_corpus, pos_dict)\nneg_k = lda_k(neg_corpus, neg_dict)"},{"cell_type":"code","execution_count":53,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"81C11CCFFACC4493B2DC48AB9B05AEFB","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"Text(0.5, 0, '负面评论LDA主题数寻优')"},"transient":{},"execution_count":53},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 720x576 with 2 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/81C11CCFFACC4493B2DC48AB9B05AEFB/q9pjk288ba.png\">"},"transient":{}}],"source":"# 绘制主题平均余弦相似度图形\nfrom matplotlib.font_manager import FontProperties  \nfont = FontProperties(size=14)\n\n\nfig = plt.figure(figsize=(10,8))\nax1 = fig.add_subplot(211)\nax1.plot(pos_k)\nax1.set_xlabel('正面评论LDA主题数寻优', fontproperties=font)\n\nax2 = fig.add_subplot(212)\nax2.plot(neg_k)\nax2.set_xlabel('负面评论LDA主题数寻优', fontproperties=font)"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"59E9522CD6414B7FA4FD2823E1C1C47C","mdEditEnable":false,"trusted":true},"source":"由图可知，对于正面评论数据，当主题数为2或3时，主题间的平均余弦相似度就达到了最低。因此，对正面评论数据做LDA，可以选择主题数为3；对于负面评论数据，当主题数为3时，主题间的平均余弦相似度也达到了最低。因此，对负面评论数据做LDA，也可以选择主题数为3。\n\n----\n\n### 评价主题分析结果\n\n根据主题数寻优结果，使用Python的Gensim模块对正面评论数据和负面评论数据分别构建LDA主题模型，设置主题数为3，经过LDA主题分析后，每个主题下生成10个最有可能出现的词语以及相应的概率"},{"cell_type":"code","execution_count":54,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"192EB1A8C23241628165744F6C39268B","collapsed":false,"scrolled":false,"trusted":true},"outputs":[],"source":"# LDA主题分析\npos_lda = models.LdaModel(pos_corpus, num_topics = 3, id2word = pos_dict)  \nneg_lda = models.LdaModel(neg_corpus, num_topics = 3, id2word = neg_dict)"},{"cell_type":"code","execution_count":55,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E5A9733E690B40CD8294878E5D5ED7C2","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"[(0,\n  '0.031*\"服务\" + 0.025*\"好评\" + 0.021*\"信赖\" + 0.020*\"售后\" + 0.019*\"人员\" + 0.016*\"太\" + 0.016*\"送\" + 0.015*\"品牌\" + 0.014*\"电话\" + 0.013*\"质量\"'),\n (1,\n  '0.029*\"很快\" + 0.028*\"不错\" + 0.026*\"值得\" + 0.023*\"客服\" + 0.017*\"物流\" + 0.017*\"差\" + 0.014*\"速度\" + 0.012*\"态度\" + 0.012*\"赞\" + 0.011*\"收到\"'),\n (2,\n  '0.115*\"安装\" + 0.050*\"满意\" + 0.038*\"师傅\" + 0.028*\"送货\" + 0.017*\"东西\" + 0.013*\"购物\" + 0.012*\"家里\" + 0.011*\"装\" + 0.010*\"真的\" + 0.010*\"预约\"')]"},"transient":{},"execution_count":55}],"source":"pos_lda.print_topics(num_words = 10)"},{"cell_type":"code","execution_count":56,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"AA161DD6FBF1484C81429292C272ADB4","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"[(0,\n  '0.022*\"东西\" + 0.019*\"装\" + 0.016*\"加热\" + 0.016*\"烧水\" + 0.015*\"漏水\" + 0.013*\"真的\" + 0.011*\"产品\" + 0.010*\"钱\" + 0.009*\"电话\" + 0.009*\"价格\"'),\n (1,\n  '0.140*\"安装\" + 0.033*\"师傅\" + 0.032*\"太\" + 0.019*\"收费\" + 0.019*\"打电话\" + 0.018*\"贵\" + 0.017*\"慢\" + 0.016*\"太慢\" + 0.012*\"材料\" + 0.011*\"高\"'),\n (2,\n  '0.031*\"垃圾\" + 0.029*\"售后\" + 0.027*\"差\" + 0.023*\"安装费\" + 0.019*\"客服\" + 0.018*\"小时\" + 0.017*\"不好\" + 0.017*\"收\" + 0.012*\"人员\" + 0.012*\"坑人\"')]"},"transient":{},"execution_count":56}],"source":"neg_lda.print_topics(num_words = 10)"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"02AC12986B954934B7875C510BF0BA4A","mdEditEnable":false,"trusted":true},"source":"## 可视化模型训练结果"},{"cell_type":"code","execution_count":63,"metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"B8E46AA74681462B989B92A1CD629FCD","collapsed":false,"scrolled":false,"trusted":true},"outputs":[{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/pyLDAvis/_prepare.py:387: DeprecationWarning: \n.ix is deprecated. Please use\n.loc for label based indexing or\n.iloc for positional indexing\n\nSee the documentation here:\nhttp://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated\n  topic_term_dists = topic_term_dists.ix[topic_order]\n","name":"stderr"},{"output_type":"execute_result","metadata":{},"data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"\n<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n\n\n<div id=\"ldavis_el761399651946959281301456176\"></div>\n<script type=\"text/javascript\">\n\nvar ldavis_el761399651946959281301456176_data = {\"mdsDat\": {\"x\": [0.13529656323544875, -0.21846012939060108, 0.08316356615515215], \"y\": [0.16476538001703206, 0.02847824359362672, -0.19324362361065875], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [34.00608444213867, 33.96202850341797, 32.031890869140625]}, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"Freq\": [268.0, 115.0, 90.0, 67.0, 66.0, 64.0, 65.0, 54.0, 62.0, 52.0, 45.0, 44.0, 42.0, 35.0, 38.0, 36.0, 41.0, 39.0, 31.0, 32.0, 33.0, 31.0, 28.0, 24.0, 26.0, 26.0, 27.0, 24.0, 25.0, 23.0, 51.50071716308594, 64.79248046875, 25.41379165649414, 24.89528465270996, 22.806486129760742, 19.00041389465332, 18.70090103149414, 17.720596313476562, 63.72024154663086, 16.140409469604492, 23.0102596282959, 26.381933212280273, 20.708276748657227, 15.497583389282227, 13.353278160095215, 14.440679550170898, 32.035953521728516, 59.706703186035156, 24.156877517700195, 19.08037567138672, 12.1773681640625, 21.496248245239258, 12.635575294494629, 25.675439834594727, 26.24259376525879, 39.175270080566406, 37.586517333984375, 10.58696174621582, 12.473925590515137, 9.892026901245117, 24.16375160217285, 13.718517303466797, 63.47084426879883, 113.10498809814453, 37.73686599731445, 260.6021728515625, 23.579824447631836, 26.0656795501709, 21.3424015045166, 30.123273849487305, 24.667110443115234, 22.35080337524414, 15.155813217163086, 14.184208869934082, 13.856162071228027, 85.60092163085938, 12.308565139770508, 15.938602447509766, 13.007131576538086, 13.738802909851074, 12.375118255615234, 10.546971321105957, 10.356136322021484, 13.750840187072754, 11.843720436096191, 9.973134994506836, 18.204347610473633, 14.718622207641602, 15.542716026306152, 9.908637046813965, 8.79604721069336, 8.469017028808594, 12.045060157775879, 17.1497859954834, 41.468231201171875, 43.642601013183594, 66.26821899414062, 34.341854095458984, 53.05464553833008, 30.355695724487305, 44.64451599121094, 22.87625503540039, 23.690744400024414, 24.239816665649414, 35.08940887451172, 22.208858489990234, 16.75434112548828, 17.27540397644043, 27.102344512939453, 18.982139587402344, 15.751397132873535, 15.751167297363281, 22.310579299926758, 13.605098724365234, 31.02967071533203, 12.089237213134766, 15.935546875, 13.2062406539917, 9.912626266479492, 9.911559104919434, 11.629328727722168, 13.262478828430176, 10.479138374328613, 11.66202163696289, 14.570035934448242, 10.004413604736328], \"Term\": [\"\\u5b89\\u88c5\", \"\\u6ee1\\u610f\", \"\\u5e08\\u5085\", \"\\u670d\\u52a1\", \"\\u5f88\\u5feb\", \"\\u9001\\u8d27\", \"\\u4e0d\\u9519\", \"\\u597d\\u8bc4\", \"\\u503c\\u5f97\", \"\\u5ba2\\u670d\", \"\\u4fe1\\u8d56\", \"\\u552e\\u540e\", \"\\u4eba\\u5458\", \"\\u9001\", \"\\u4e1c\\u897f\", \"\\u592a\", \"\\u7269\\u6d41\", \"\\u5dee\", \"\\u7535\\u8bdd\", \"\\u54c1\\u724c\", \"\\u901f\\u5ea6\", \"\\u8d2d\\u7269\", \"\\u8d28\\u91cf\", \"\\u670d\\u52a1\\u6001\\u5ea6\", \"\\u7b2c\\u4e8c\\u5929\", \"\\u5bb6\\u91cc\", \"\\u6001\\u5ea6\", \"\\u544a\\u8bc9\", \"\\u5c0f\\u65f6\", \"\\u6d3b\\u52a8\", \"\\u5ba2\\u670d\", \"\\u5f88\\u5feb\", \"\\u7b2c\\u4e8c\\u5929\", \"\\u5c0f\\u65f6\", \"\\u65f6\\u95f4\", \"\\u574f\", \"\\u53ea\\u80fd\", \"\\u8d27\", \"\\u4e0d\\u9519\", \"\\u8d39\\u7528\", \"\\u4ea7\\u54c1\", \"\\u6001\\u5ea6\", \"\\u611f\\u89c9\", \"\\u8001\\u4eba\", \"\\u60f3\", \"\\u6c34\\u9600\", \"\\u901f\\u5ea6\", \"\\u503c\\u5f97\", \"\\u5feb\\u9012\", \"\\u529b\", \"\\u914d\\u9001\", \"\\u4e0a\\u95e8\", \"\\u4e24\\u4e2a\", \"\\u6536\\u5230\", \"\\u8d5e\", \"\\u7269\\u6d41\", \"\\u5dee\", \"\\u94b1\", \"\\u70e7\\u6c34\", \"\\u8d27\\u7269\", \"\\u8bc4\", \"\\u5c11\", \"\\u9001\\u8d27\", \"\\u6ee1\\u610f\", \"\\u4e1c\\u897f\", \"\\u5b89\\u88c5\", \"\\u771f\\u7684\", \"\\u5bb6\\u91cc\", \"\\u6536\", \"\\u8d2d\\u7269\", \"\\u88c5\", \"\\u9884\\u7ea6\", \"\\u6c9f\\u901a\", \"\\u627e\", \"\\u6c34\", \"\\u5e08\\u5085\", \"\\u6c34\\u7ba1\", \"\\u82b1\", \"\\u5bb6\\u7535\", \"\\u652f\\u6301\", \"\\u5730\\u65b9\", \"\\u8d1f\\u8d23\", \"\\u665a\\u4e0a\", \"\\u8d2d\\u4e70\", \"\\u5916\\u9762\", \"\\u6d17\\u6fa1\", \"\\u6253\\u7535\\u8bdd\", \"\\u4ef7\\u683c\", \"\\u7279\\u522b\", \"\\u8f9b\\u82e6\", \"\\u661f\", \"\\u4e0d\\u5230\", \"\\u5b89\\u88c5\\u8d39\", \"\\u771f\\u5fc3\", \"\\u4eba\\u5458\", \"\\u552e\\u540e\", \"\\u670d\\u52a1\", \"\\u9001\", \"\\u597d\\u8bc4\", \"\\u7535\\u8bdd\", \"\\u4fe1\\u8d56\", \"\\u6d3b\\u52a8\", \"\\u544a\\u8bc9\", \"\\u670d\\u52a1\\u6001\\u5ea6\", \"\\u592a\", \"\\u9001\\u5230\", \"\\u60c5\\u51b5\", \"\\u5bb6\", \"\\u8d28\\u91cf\", \"\\u70b9\", \"\\u53d1\\u73b0\", \"\\u5356\", \"\\u559c\\u6b22\", \"\\u6df7\", \"\\u54c1\\u724c\", \"\\u6d17\", \"\\u6536\\u8d39\", \"\\u5230\\u8d27\", \"\\u7406\\u89e3\", \"\\u914d\\u4ef6\", \"\\u4e13\\u4e1a\", \"\\u7c73\", \"\\u5929\", \"\\u633a\\u5feb\", \"\\u8bc4\\u4ef7\", \"\\u8dd1\"], \"Total\": [268.0, 115.0, 90.0, 67.0, 66.0, 64.0, 65.0, 54.0, 62.0, 52.0, 45.0, 44.0, 42.0, 35.0, 38.0, 36.0, 41.0, 39.0, 31.0, 32.0, 33.0, 31.0, 28.0, 24.0, 26.0, 26.0, 27.0, 24.0, 25.0, 23.0, 52.41262435913086, 66.27510833740234, 26.013662338256836, 25.551681518554688, 23.47651481628418, 19.605180740356445, 19.345619201660156, 18.33875846862793, 65.97252655029297, 16.73174285888672, 23.893882751464844, 27.396421432495117, 21.505430221557617, 16.112695693969727, 13.903175354003906, 15.040541648864746, 33.37383270263672, 62.274871826171875, 25.221355438232422, 19.939817428588867, 12.740654945373535, 22.510107040405273, 13.233492851257324, 26.910123825073242, 27.515148162841797, 41.14495849609375, 39.489253997802734, 11.172134399414062, 13.17559814453125, 10.49355411529541, 25.69249725341797, 14.600815773010254, 64.59589385986328, 115.85872650146484, 38.82763671875, 268.1905822753906, 24.32331085205078, 26.97955894470215, 22.099966049194336, 31.199697494506836, 25.605810165405273, 23.20380401611328, 15.824963569641113, 14.853646278381348, 14.52244758605957, 90.00812530517578, 12.97183895111084, 16.834997177124023, 13.740782737731934, 14.520991325378418, 13.092432975769043, 11.184060096740723, 11.000541687011719, 14.608663558959961, 12.592105865478516, 10.649825096130371, 19.444555282592773, 15.727330207824707, 16.6092472076416, 10.599040031433105, 9.424239158630371, 9.092089653015137, 12.985152244567871, 18.628402709960938, 42.20248794555664, 44.445716857910156, 67.5169677734375, 35.004459381103516, 54.15494918823242, 31.021665573120117, 45.655330657958984, 23.501298904418945, 24.346385955810547, 24.924800872802734, 36.16932678222656, 23.026390075683594, 17.387317657470703, 17.93987464904785, 28.164413452148438, 19.82498550415039, 16.46947479248047, 16.46947479248047, 23.33042335510254, 14.236698150634766, 32.4886474609375, 12.68487548828125, 16.73857879638672, 13.915935516357422, 10.486284255981445, 10.485824584960938, 12.314751625061035, 14.04740047454834, 11.11204719543457, 12.398553848266602, 15.501480102539062, 10.715566635131836], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.0611000061035156, 1.055999994277954, 1.055299997329712, 1.0526000261306763, 1.0497000217437744, 1.0472999811172485, 1.044700026512146, 1.0442999601364136, 1.0439000129699707, 1.0426000356674194, 1.0408999919891357, 1.0408999919891357, 1.0408999919891357, 1.0397000312805176, 1.0383000373840332, 1.0378999710083008, 1.0377000570297241, 1.0364999771118164, 1.0355000495910645, 1.034600019454956, 1.033400058746338, 1.0325000286102295, 1.0324000120162964, 1.0317000150680542, 1.0312999486923218, 1.0296000242233276, 1.0291999578475952, 1.0247999429702759, 1.023900032043457, 1.0196000337600708, 1.017300009727478, 1.0162999629974365, 1.062399983406067, 1.055899977684021, 1.0513999462127686, 1.051200032234192, 1.0489000082015991, 1.0455000400543213, 1.0449999570846558, 1.044800043106079, 1.0426000356674194, 1.0425000190734863, 1.0367000102996826, 1.0338000059127808, 1.0329999923706055, 1.0297000408172607, 1.027400016784668, 1.0252000093460083, 1.0250999927520752, 1.0246000289916992, 1.0235999822616577, 1.021299958229065, 1.0196000337600708, 1.0194000005722046, 1.0187000036239624, 1.014299988746643, 1.0140000581741333, 1.0135999917984009, 1.0135999917984009, 1.0125999450683594, 1.0109000205993652, 1.0089000463485718, 1.0047999620437622, 0.9972000122070312, 1.12090003490448, 1.1202000379562378, 1.1197999715805054, 1.1193000078201294, 1.117900013923645, 1.1167000532150269, 1.1160000562667847, 1.1115000247955322, 1.1110999584197998, 1.1105999946594238, 1.1081000566482544, 1.1023000478744507, 1.1014000177383423, 1.1007000207901, 1.100000023841858, 1.0950000286102295, 1.0938999652862549, 1.0937999486923218, 1.0937000513076782, 1.0930999517440796, 1.0924999713897705, 1.0902999639511108, 1.0893000364303589, 1.0860999822616577, 1.082200050354004, 1.0821000337600708, 1.0812000036239624, 1.080899953842163, 1.079800009727478, 1.0772000551223755, 1.0765000581741333, 1.0698000192642212], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.785599946975708, -3.555999994277954, -4.4918999671936035, -4.512499809265137, -4.600100040435791, -4.782700061798096, -4.798600196838379, -4.852499961853027, -3.572700023651123, -4.945899963378906, -4.591300010681152, -4.454500198364258, -4.696700096130371, -4.986499786376953, -5.13539981842041, -5.05709981918335, -4.260300159454346, -3.637700080871582, -4.542600154876709, -4.778500080108643, -5.22760009765625, -4.659299850463867, -5.190700054168701, -4.4816999435424805, -4.459799766540527, -4.059100151062012, -4.100500106811523, -5.367599964141846, -5.20359992980957, -5.435500144958496, -4.542300224304199, -5.108399868011475, -3.5752999782562256, -2.9976000785827637, -4.095300197601318, -2.162899971008301, -4.565499782562256, -4.4653000831604, -4.665200233459473, -4.3206000328063965, -4.520400047302246, -4.61899995803833, -5.007500171661377, -5.073800086975098, -5.0971999168396, -3.27620005607605, -5.21560001373291, -4.957200050354004, -5.160399913787842, -5.1057000160217285, -5.21019983291626, -5.370100021362305, -5.388299942016602, -5.104800224304199, -5.2540998458862305, -5.426000118255615, -4.82420015335083, -5.036799907684326, -4.9822998046875, -5.432499885559082, -5.551599979400635, -5.5894999504089355, -5.237199783325195, -4.883900165557861, -3.942500114440918, -3.891400098800659, -3.4737000465393066, -4.13100004196167, -3.6960999965667725, -4.25439977645874, -3.8687000274658203, -4.537300109863281, -4.502299785614014, -4.479400157928467, -4.109499931335449, -4.56689977645874, -4.848700046539307, -4.8180999755859375, -4.367800235748291, -4.723899841308594, -4.9105000495910645, -4.9105000495910645, -4.562300205230713, -5.0569000244140625, -4.232399940490723, -5.175099849700928, -4.898799896240234, -5.086699962615967, -5.373600006103516, -5.373700141906738, -5.213900089263916, -5.082399845123291, -5.317999839782715, -5.210999965667725, -4.988399982452393, -5.3643999099731445]}, \"token.table\": {\"Topic\": [1, 3, 2, 1, 2, 3, 3, 1, 2, 3, 1, 1, 3, 1, 2, 1, 3, 1, 2, 3, 3, 1, 3, 3, 1, 3, 1, 2, 3, 3, 1, 3, 2, 1, 2, 3, 1, 3, 1, 2, 3, 1, 2, 3, 1, 2, 1, 3, 2, 2, 1, 1, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 3, 1, 2, 3, 1, 1, 1, 2, 2, 3, 2, 2, 1, 2, 3, 3, 1, 2, 2, 1, 2, 3, 3, 2, 2, 1, 2, 3, 2, 3, 3, 1, 2, 3, 3, 1, 1, 2, 3, 1, 2, 3, 3, 3, 1, 2, 2, 1, 3, 1, 1, 2, 1, 2, 1, 2, 3, 1, 3, 2, 1, 1, 1, 3, 2, 1, 2, 1, 1, 2, 3, 3, 2, 3, 3, 1, 2, 3, 1, 2, 3, 3, 1, 1, 2], \"Freq\": [0.9329142570495605, 0.04442448914051056, 0.879885733127594, 0.9701007604598999, 0.015157824382185936, 0.015157824382185936, 0.9744411110877991, 0.02575485035777092, 0.9786843657493591, 0.02575485035777092, 0.9823558926582336, 0.9625895023345947, 0.9715067148208618, 0.06358358263969421, 0.9537537693977356, 0.021903246641159058, 0.98564612865448, 0.963470458984375, 0.01605784147977829, 0.01605784147977829, 0.9341807961463928, 0.952867329120636, 0.9714942574501038, 0.9714942574501038, 0.9821345210075378, 0.9857726097106934, 0.03077998198568821, 0.03077998198568821, 0.9541794657707214, 0.9899716377258301, 0.04286248981952667, 0.9429747462272644, 0.9165599942207336, 0.969131588935852, 0.9529780149459839, 0.8999241590499878, 0.02764773555099964, 0.967670738697052, 0.01846553385257721, 0.01846553385257721, 0.9786732196807861, 0.014914766885340214, 0.9731885194778442, 0.011186075396835804, 0.07701103389263153, 0.9241324067115784, 0.9921273589134216, 0.9476097226142883, 0.9460887312889099, 0.9636925458908081, 0.9784091711044312, 0.9588505625724792, 0.9622871279716492, 0.02532334439456463, 0.02532334439456463, 0.022220216691493988, 0.9554693102836609, 0.022220216691493988, 0.980760395526886, 0.015088621526956558, 0.015088621526956558, 0.9515745639801025, 0.03964893892407417, 0.9490290284156799, 0.036501117050647736, 0.9777241349220276, 0.9350382089614868, 0.9764975309371948, 0.05142827704548836, 0.9257090091705322, 0.9425294995307922, 0.9678547978401184, 0.9641215205192566, 0.950227677822113, 0.9661791324615479, 0.037160735577344894, 0.037160735577344894, 0.9558756351470947, 0.9797024726867676, 0.9549842476844788, 0.9090461730957031, 0.014811092056334019, 0.014811092056334019, 0.9775320291519165, 0.9628963470458984, 0.9640248417854309, 0.9250808358192444, 0.9308175444602966, 0.9478694796562195, 0.9460085034370422, 0.9389825463294983, 0.978669285774231, 0.9833740592002869, 0.008631201460957527, 0.9753257632255554, 0.008631201460957527, 0.9583866000175476, 0.9107745885848999, 0.9478682279586792, 0.024304313585162163, 0.024304313585162163, 0.06020742282271385, 0.9633187651634216, 0.06020742282271385, 0.9536266326904297, 0.9670660495758057, 0.05368146672844887, 0.9125849604606628, 0.9867078065872192, 0.9610334634780884, 0.9254381060600281, 0.9309428930282593, 0.05940007045865059, 0.9504011273384094, 0.03905363753437996, 0.9763408899307251, 0.9341248273849487, 0.03892187029123306, 0.03892187029123306, 0.06450996547937393, 0.967649519443512, 0.9835426211357117, 0.9815277457237244, 0.9529659748077393, 0.03550579771399498, 0.9586565494537354, 0.9583354592323303, 0.03205159306526184, 0.9615477919578552, 0.9562661647796631, 0.9449340105056763, 0.036343615502119064, 0.036343615502119064, 0.9332217574119568, 0.9434816837310791, 0.9713048338890076, 0.955425500869751, 0.015480860136449337, 0.9752941727638245, 0.015480860136449337, 0.9588350057601929, 0.029963593930006027, 0.029963593930006027, 0.9536684155464172, 0.9418668150901794, 0.9845924973487854, 0.9481204152107239], \"Term\": [\"\\u4e0a\\u95e8\", \"\\u4e0a\\u95e8\", \"\\u4e0d\\u5230\", \"\\u4e0d\\u9519\", \"\\u4e0d\\u9519\", \"\\u4e0d\\u9519\", \"\\u4e13\\u4e1a\", \"\\u4e1c\\u897f\", \"\\u4e1c\\u897f\", \"\\u4e1c\\u897f\", \"\\u4e24\\u4e2a\", \"\\u4ea7\\u54c1\", \"\\u4eba\\u5458\", \"\\u4ef7\\u683c\", \"\\u4ef7\\u683c\", \"\\u4fe1\\u8d56\", \"\\u4fe1\\u8d56\", \"\\u503c\\u5f97\", \"\\u503c\\u5f97\", \"\\u503c\\u5f97\", \"\\u5230\\u8d27\", \"\\u529b\", \"\\u5356\", \"\\u53d1\\u73b0\", \"\\u53ea\\u80fd\", \"\\u544a\\u8bc9\", \"\\u54c1\\u724c\", \"\\u54c1\\u724c\", \"\\u54c1\\u724c\", \"\\u552e\\u540e\", \"\\u559c\\u6b22\", \"\\u559c\\u6b22\", \"\\u5730\\u65b9\", \"\\u574f\", \"\\u5916\\u9762\", \"\\u5929\", \"\\u592a\", \"\\u592a\", \"\\u597d\\u8bc4\", \"\\u597d\\u8bc4\", \"\\u597d\\u8bc4\", \"\\u5b89\\u88c5\", \"\\u5b89\\u88c5\", \"\\u5b89\\u88c5\", \"\\u5b89\\u88c5\\u8d39\", \"\\u5b89\\u88c5\\u8d39\", \"\\u5ba2\\u670d\", \"\\u5bb6\", \"\\u5bb6\\u7535\", \"\\u5bb6\\u91cc\", \"\\u5c0f\\u65f6\", \"\\u5c11\", \"\\u5dee\", \"\\u5dee\", \"\\u5dee\", \"\\u5e08\\u5085\", \"\\u5e08\\u5085\", \"\\u5e08\\u5085\", \"\\u5f88\\u5feb\", \"\\u5f88\\u5feb\", \"\\u5f88\\u5feb\", \"\\u5feb\\u9012\", \"\\u5feb\\u9012\", \"\\u6001\\u5ea6\", \"\\u6001\\u5ea6\", \"\\u60c5\\u51b5\", \"\\u60f3\", \"\\u611f\\u89c9\", \"\\u6253\\u7535\\u8bdd\", \"\\u6253\\u7535\\u8bdd\", \"\\u627e\", \"\\u633a\\u5feb\", \"\\u652f\\u6301\", \"\\u6536\", \"\\u6536\\u5230\", \"\\u6536\\u5230\", \"\\u6536\\u5230\", \"\\u6536\\u8d39\", \"\\u65f6\\u95f4\", \"\\u661f\", \"\\u665a\\u4e0a\", \"\\u670d\\u52a1\", \"\\u670d\\u52a1\", \"\\u670d\\u52a1\", \"\\u670d\\u52a1\\u6001\\u5ea6\", \"\\u6c34\", \"\\u6c34\\u7ba1\", \"\\u6c34\\u9600\", \"\\u6c9f\\u901a\", \"\\u6d17\", \"\\u6d17\\u6fa1\", \"\\u6d3b\\u52a8\", \"\\u6df7\", \"\\u6ee1\\u610f\", \"\\u6ee1\\u610f\", \"\\u6ee1\\u610f\", \"\\u70b9\", \"\\u70e7\\u6c34\", \"\\u7269\\u6d41\", \"\\u7269\\u6d41\", \"\\u7269\\u6d41\", \"\\u7279\\u522b\", \"\\u7279\\u522b\", \"\\u7279\\u522b\", \"\\u7406\\u89e3\", \"\\u7535\\u8bdd\", \"\\u771f\\u5fc3\", \"\\u771f\\u5fc3\", \"\\u771f\\u7684\", \"\\u7b2c\\u4e8c\\u5929\", \"\\u7c73\", \"\\u8001\\u4eba\", \"\\u82b1\", \"\\u82b1\", \"\\u88c5\", \"\\u88c5\", \"\\u8bc4\", \"\\u8bc4\", \"\\u8bc4\", \"\\u8bc4\\u4ef7\", \"\\u8bc4\\u4ef7\", \"\\u8d1f\\u8d23\", \"\\u8d27\", \"\\u8d27\\u7269\", \"\\u8d28\\u91cf\", \"\\u8d28\\u91cf\", \"\\u8d2d\\u4e70\", \"\\u8d2d\\u7269\", \"\\u8d2d\\u7269\", \"\\u8d39\\u7528\", \"\\u8d5e\", \"\\u8d5e\", \"\\u8d5e\", \"\\u8dd1\", \"\\u8f9b\\u82e6\", \"\\u9001\", \"\\u9001\\u5230\", \"\\u9001\\u8d27\", \"\\u9001\\u8d27\", \"\\u9001\\u8d27\", \"\\u901f\\u5ea6\", \"\\u901f\\u5ea6\", \"\\u901f\\u5ea6\", \"\\u914d\\u4ef6\", \"\\u914d\\u9001\", \"\\u94b1\", \"\\u9884\\u7ea6\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [2, 3, 1]};\n\nfunction LDAvis_load_lib(url, callback){\n  var s = document.createElement('script');\n  s.src = url;\n  s.async = true;\n  s.onreadystatechange = s.onload = callback;\n  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n  document.getElementsByTagName(\"head\")[0].appendChild(s);\n}\n\nif(typeof(LDAvis) !== \"undefined\"){\n   // already loaded: just create the visualization\n   !function(LDAvis){\n       new LDAvis(\"#\" + \"ldavis_el761399651946959281301456176\", ldavis_el761399651946959281301456176_data);\n   }(LDAvis);\n}else if(typeof define === \"function\" && define.amd){\n   // require.js is available: use it to load d3/LDAvis\n   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n   require([\"d3\"], function(d3){\n      window.d3 = d3;\n      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n        new LDAvis(\"#\" + \"ldavis_el761399651946959281301456176\", ldavis_el761399651946959281301456176_data);\n      });\n    });\n}else{\n    // require.js not available: dynamically load d3 & LDAvis\n    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n                 new LDAvis(\"#\" + \"ldavis_el761399651946959281301456176\", ldavis_el761399651946959281301456176_data);\n            })\n         });\n}\n</script>"},"transient":{},"execution_count":63}],"source":"import pyLDAvis\n\nvis = pyLDAvis.gensim.prepare(pos_lda,pos_corpus,pos_dict)\n# 需要的三个参数都可以从硬盘读取的，前面已经存储下来了\n\n# 在浏览器中心打开一个界面\n# pyLDAvis.show(vis)\n\n# 在notebook的output cell中显示\npyLDAvis.display(vis)"},{"cell_type":"markdown","metadata":{"jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"EF068BC64BA74280B4D54E3AB1D1D296","mdEditEnable":false,"trusted":true},"source":"综合以上对主题及其中的高频特征词的分析得出，美的电热水器有价格实惠、性价比高、外观好看、服务好等优势。相对而言，用户对美的电热水器的抱怨点主要体现在安装的费用高及售后服务差等方面。因此，用户的购买原因可以总结为以下几个方面：美的是大品牌值得信赖、美的电热水器价格实惠、性价比高。\n\n根据对京东平台上美的电热水器的用户评价情况进行LDA主题模型分析，对美的品牌提出以下两点建议：\n1. 在保持热水器使用方便、价格实惠等优点的基础上，对热水器进行加热功能上的改进，从整体上提升热水器的质量。\n2. 提升安装人员及客服人员的整体素质，提高服务质量，注重售后服务。建立安装费用收取的明文细则，并进行公布，以减少安装过程中乱收费的现象。适度降低安装费用和材料费用，以此在大品牌的竞争中凸显优势。\n\n## 参考资料\n[Python数据分析与挖掘实战](https://book.douban.com/subject/34888317/)"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":2}